Naval  Submarine  Medical  Research  Laboratory 

NSMRL  REPORT  1206  06  JUNE  1997 


RELIABILITY  OF  THE  SUBSCREEN 
PSYCHOLOGICAL  SCREENING  INVENTORY 


Douglas  W.  Theriaque,  MS. 
Christine  Schlichting,  PhD. 


Released  by 

R.  G.  Walter,  CAPT,  DC,  USN 
Commanding  Officer 

Naval  Submarine  Medical  Research  Laboratory 


Approved  for  public  release;  distribution  unlimited 


Reliability  of  the  SUBSCREEN 
Psychological  Screening  Inventory 


Douglas  W.  Theriaque,  MS 
Christine  Schlichting,  Ph.D. 


Naval  Submarine  Medical  Research  Laboratory 
Report  1206 


Naval  Sea  Systems  Command  Task 

Naval  Medical  Research  and  Development  Command 
Research  Work  Unit  -  5403 


Approved  and  released  by 

R.  G.  Walter,  CAPT,  DC,  USN 
Commanding  Officer 
N  avSubMedRschLab 


Approved  for  public  release;  distributed  unlimited 


SUMMARY  PAGE 


THE  PROBLEM 

SUBSCREEN  is  a  240  item  psychological  inventory  used  to  evaluate  prospective 
submariners'  ability  to  adjust  to  Naval  Submarine  School  (NAVSUBSCOL)  and  subsequent 
submarine  service,  and  refer  students  with  atypical  results  for  mental  health  interviews.  With  any 
screening  tool  reliability  of  the  procedure  is  always  an  important  issue.  Results  of  a  1990  study 
by  Katz  and  Rexer  indicated  that  several  SUBSCREEN  subscales  had  low  reliability.  Some  of 
the  items  in  these  subscales  had  an  unusually  high  number  of  neutral  or  missing  responses,  while 
others  demonstrated  negative  intercorrelations  with  the  subscale  to  which  they  belonged. 
Additionally,  the  means  and  standard  deviations  currently  being  used  to  calculate  subscale 
standardized  scores  needed  to  be  reevaluated  to  ensure  consistent  and  accurate  SUBSCREEN 
results. 

THE  FINDINGS 

All  SUBSCREEN  subscales  were  found  to  have  moderate  to  high  reliability  (Cronbach’s 
[1951]  alpha  from  .65  to  .87)  after  correcting  the  scoring  of  the  items  that  correlated  negatively 
with  their  respective  subscales.  The  continued  existence  of  items  with  a  high  number  of  neutral 
or  missing  responses  was  confirmed  and  implementation  of  the  new  means  and  standard 
deviations  was  recommended  based  on  a  renorming  procedure. 

THE  APPLICATION 

Improved  accuracy  and  consistency  of  SUBSCREEN  results. 


ADMINISTRATIVE  INFORMATION 

This  investigation  was  conducted  under  Naval  Medical  Research  and  Development  Command 
Research  Work  Unit  5403,  "Psychological  screening  and  testing  in  support  of  submarine 
training,”  in  response  to  a  Naval  Sea  Systems  Command  Work  Request.  The  views  expressed  in 
this  report  are  those  of  the  authors  and  do  not  reflect  the  official  policy  or  position  of  the 
Department  of  the  Navy,  Department  of  Defense,  or  the  U.S.  government.  This  report  was 
approved  for  publication  on  06  Jun  1997  and  designated  Naval  Submarine  Medical  Research 
Laboratory  Report  1206. 
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ABSTRACT 


SUBSCREEN  is  a  psychological  screening  tool  used  at  the  Naval  Submarine  School 
(NAVSUBSCOL)  to  assess  the  ability  of  prospective  submariners  to  adjust  to  both 
NAVSUBSCOL  and  subsequent  submarine  service.  Standardized  scores  are  calculated  for  each 
student  for  each  of  the  27  subscales  to  determine  how  the  student’s  scores  compare  to  a  group  of 
a  large  number  of  previously  tested  NAVSUBSCOL  students.  An  unpublished  reliability 
analysis  by  Katz  and  Rexer  (1990)  suggested  low  reliabilities  of  several  SUBSCREEN  subscales 
and  identified  the  items  that  contributed  to  the  low  subscale  reliabilities.  This  early  work  had  a 
relatively  small  number  of  test  results  in  the  comparison  group  and  the  authors  realized  the  need 
to  provide  a  more  definitive  analysis  of  the  SUBSCREEN's  reliability  in  order  to  recommend 
changes  to  certain  subscales  and  items. 

A  reliability  study  similar  to  Katz  and  Rexer’s  was  conducted  using  a  larger  data  set 
containing  19,517  observations  to  verify  their  findings.  Initial  results  closely  resembled  Katz  & 
Rexer’s  earlier  work.  Two  types  of  problems  were  found  in  certain  items:  some  were  scored 
incorrectly,  and  some  yielded  an  unusual  number  of  missing  or  neutral  responses.  A  corrected 
scoring  procedure  was  instituted  for  items  in  the  first  category.  Review  and  possible  elimination 
of  the  items  yielding  missing  or  neutral  responses  is  ongoing.  Following  corrections  in  the 
scoring  procedure,  the  reliability  analysis  was  performed  again  to  re-examine  the  subscales’ 
reliability.  The  reliability  of  the  three  subscales  demonstrating  low  reliability  in  the  earlier  study 
improved  greatly  after  scoring  inconsistencies  were  corrected.  Means  and  standard  deviations 
were  obtained  from  the  larger  database  for  25  of  the  subscales  and  the  new  values  are  now 
included  in  the  scoring  process. 
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Reliability  of  the  SUBSCREEN  Psychological  Screening  Inventory 


Submarine  service  involves  prolonged 
periods  of  time  underwater  and  little,  if  any, 
contact  with  the  outside  world.  This, 
combined  with  the  fact  that  psychiatric  help 
is  unavailable  while  onboard,  makes  the 
identification  of  individuals  with  mental 
health  problems  extremely  important.  The 
identification  and  exclusion  of  such 
individuals  assures  that  the  mental  health  of 
submariners  is  of  the  highest  quality.  This, 
in  turn,  helps  to  assure  crew  safety  as  well  as 
improved  general  efficiency  and  cost- 
effectiveness  for  the  Navy. 

The  SUBSCREEN  testing  process  is 
used  to  identify  candidates  that  are  potenti¬ 
ally  unsuitable  for  Naval  Submarine  School 
(NAVSUBSCOL)  and/or  submarine  service 
because  of  psychological  or  motivational 
difficulties.  Those  identified  are  referred  to 
the  Psychiatry  Department  of  the  Naval 
Hospital  on  the  submarine  base  Groton,  CT 
for  a  mental  health  status  interview. 

Developed  in  1986,  the  SUB- 
SCREEN’s  240  items  comprise  27  inde¬ 
pendent  subscales  and  procedural  scores. 
Procedural  scores  are  subscales  calculated 
through  the  use  of  composite  scores. 
Composite  scores  are  values  that  are  derived 
either  by  combining  several  subscale  values 
or  by  examining  subjects’  response  patterns 
(e.g.,  the  number  of  neutral  responses). 
Procedural  scores  typically  measure  con¬ 
structs  not  overtly  evaluated  by  question 
content.  For  instance,  SUBSCREEN 
contains  an  Extreme  subscale  that  examines 
a  student’s  response  pattern  to  determine  the 
frequency  with  which  he  is  responding 
“Strongly  Agree”  or  “Strongly  Disagree.” 

The  inventory  is  administered  to  all 
Basic  Enlisted  Submarine  School  (BESS) 
students  during  their  first  week  of  training  at 
the  Naval  Submarine  Base  in  Groton,  CT. 


The  scoring  algorithm  generates  referrals 
based  on  the  scores  of  14  of  the  27 
subscales.  These  14  subscales  relate  to 
Diagnostic  and  Statistical  Manual  (DSM-IV) 
(1994)  psychological  dimensions  and/or 
reflect  conditions  which  make  an  individual 
unacceptable  for  submarine  service  (e.g. 
claustrophobia,  suicidal  ideation,  sleep 
walking)  according  to  the  Manual  of  the 
Medical  Department  (1992).  The  remaining 
13  subscales  relate  to  non-classifiable 
dimensions  (primarily  motivation).  The 
scores  from  these  subscales,  which  do  not 
automatically  generate  referrals,  are  also 
considered  by  the  Naval  Submarine  Medical 
Research  Laboratory  (NSMRL)  examiner 
when  making  the  final  determination 
regarding  referrals.  The  reader  is  referred  to 
Schlichting  (1993)  for  a  comprehensive 
discussion  and  review  of  the  SUBSCREEN 
inventory. 

Of  all  enlisted  students  tested  between 
1988  and  1995,  approximately  9.7  percent 
were  referred  to  the  Psychiatry  Department; 
of  those  referred,  75  percent  were  returned 
to  BESS,  9.1  percent  were  transferred  to  the 
surface  fleet,  and  15.9  percent  were 
discharged  from  the  Navy  (Schlichting, 
1993). 

In  the  construction  and  maintenance  of 
any  screening  tool  it  is  essential  that  the  tool 
demonstrate  internal  reliability  and  that  new 
norms  be  developed  as  the  number  of  scores 
available  increases  and  substantial  time 
passes  (Cronbach,  1990).  In  1990,  Katz  and 
Rexer  examined  the  internal  consistency  of 
25  of  the  27  SUBSCREEN  subscales,1 
renormed  the  instrument,  and  identified 


The  two  subscales  not  included  in  the  analysis 
are  the  procedural  subscales.  Reliability  analysis  is 
not  appropriate  for  these  composite  subscales. 
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problem  items.  While  this  investigation 
found  that  most  of  the  25  subscales  demon¬ 
strated  high  levels  of  reliability  (alpha  >.70), 
the  Self  Critical,  Problems  Submarine 
School,  and  Uncertain  About  Submarines 
subscales  displayed  moderate  or  low 
reliability.  Katz  and  Rexer  recommended 
that  the  subscales  demonstrating  inadequate 
reliability  be  re-examined  and  that  some  of 
the  items  be  re-worded  or  deleted  to  improve 
the  subscales'  internal  consistency. 

As  part  of  a  renorming  procedure,  an 
analysis  of  the  means  and  standard 
deviations  indicated  that  the  newly 
calculated  values  (based  on  6,721  subjects) 
"failed  to  closely  match  the  original  sample's 
subscale  means  and  standard  deviations" 
(Katz  &  Rexer,  Unpublished,  1990,  p.  6) 
calculated  in  1986  and  based  on  a  sample  of 
approximately  800  subjects.  Upon  Katz  and 
Rexer's  recommendation,  the  new  means 
and  standard  deviations  were  implemented 
in  1990.  Of  the  fourteen  problem  items 
identified  in  the  study,  five  were  so 
classified  because  of  a  large  number  of 
neutral  or  missing  responses  (450  or  more 
[=>  6.70  %]).  The  remaining  nine  problem 
items  were  found  to  have  negative  item-total 
corrected  correlations  within  their  respective 
subscales.  The  scoring  algorithm  was 
subsequently  adjusted  to  exclude  one  of 
these  nine  items  from  the  computation  of  its 
subscale  (Uncertain  About  Submarines) 
score;  however,  the  item  was  left  in  the 
inventory.  While  identification  of  these 
fourteen  problem  items  was  performed  by 
Katz  and  Rexer,  with  the  exception  of  the 
one  item  listed  above,  examination  of  the 
specific  content  of  each  question,  within  the 
context  of  its  subscale,  was  not  conducted. 

The  purpose  of  the  present  investiga¬ 
tion  was  to  perform  a  reliability  analysis 
similar  to  Katz  and  Rexer’s  to  verify  their 
earlier  findings.  If  the  findings  indicated 


that  the  problems  they  identified  still  exist, 
the  problematic  items  and  the  scoring  rou¬ 
tine  would  be  examined  to  determine  where 
the  difficulty  lies.  If  the  scoring  routine  was 
identified  as  the  source  of  the  problem  it 
would  be  modified  to  correct  for  the  nega¬ 
tive  correlations.  The  reliability  analysis 
would  then  be  re-run  to:  1)  determine  the 
internal  consistency  of  the  subscales;  2) 
determine  which  specific  SUBSCREEN 
items  yield  disproportionate  quantities  of 
neutral  responses  or  responses  that  do  not 
correlate  well  with  the  other  items  in  their 
respective  subscales;  3)  revise  the  means 
and  standard  deviations  to  reflect  the  larger 
number  of  student  scores  now  available. 

Methods  and  Materials 
Participants 

Subjects  consisted  of  19,517  BESS 
students  enrolled  in  training  from  1988 
through  1995.  The  median  age  of  these 
males  was  20.0  years,  and  the  median 
educational  grade  level  was  12.0.  Class 
sizes  varied,  ranging  from  10  to  181 
subjects,  and  approximately  350  classes 
were  tested  over  the  seven  year  period. 

Design  and  Procedure 

SUBSCREEN  testing  of  BESS 
students  typically  occurred  during  the  first 
week  of  a  five  week  training  course.  Test 
administration  usually  took  place  each 
Monday,  beginning  at  0730  and  ending  by 
0900.  As  part  of  the  consent  process, 
students  were  informed  of  the  purpose  of  the 
test  and  the  percentage  of  students  typically 
referred  for  further  evaluation.  They  were 
also  informed  that  psychological  screening 
via  this  test  is  a  requirement  of 
NAVSUBSCOL;  however,  students  are  not 
required  to  provide  an  identifying  social 
security  number.  After  the  introduction  and 
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reading  the  Privacy  Act  statement  aloud,  any 
student’s  questions  were  answered.  Finally, 
students  signed  consent-to-participate  and 
privacy  act  forms. 

Data  Scoring/Handling 

Standard  Trans-Optic®  (National 
Computer  Systems  Inc.,  Columbia,  PA) 
computer  scan  sheets  were  used  to  obtain 
each  student's  responses  to  the  240  items. 
Subjects  rated  how  closely  they  agreed  with 
each  item  by  selecting  one  of  five  possible 
responses:  "A"  Strongly  Agree;  "B"  Agree; 
"C"  Neutral;  "D"  Disagree;  or  "E"  Strongly 
Disagree.  Upon  completion  of  the  testing, 
response  sheets  were  scanned  and  computer 
scored  by  an  algorithm  that  computed  the  27 
standardized  subscales  and  identified  those 
individuals  whose  scores  met  or  exceed 
referral  criteria.  The  resulting  computerized 
data  and  output  files  were  read  into  a 
Statistical  Package  for  the  Social  Sciences 
(SPSS)  data  file.  All  statistics  were 
computed  using  SPSS  for  Windows,  version 
6.1  (SPSS,  Chicago,  IL.). 

Accuracy  of  the  data  was  ensured  by 
examining  the  data  in  a  number  of  ways. 

The  process  began  with  a  class  by  class 
verification  of  the  number  of  response  sheets 
and  matching  electronic  records.  Addition¬ 
ally,  all  social  security  numbers,  dates  of 
birth,  and  BESS  class  numbers  were 
checked  for  accuracy,  and,  in  cases  where 
inconsistencies  occurred,  corrected  by 
referring  to  the  hardcopy.  Class  rosters 
provided  by  BESS  were  also  used  to  resolve 
discrepancies  in  this  demographic  data. 
Finally,  random  samples  of  data  (the  240 
raw  item  scores  and  the  standardized 
subscale  scores)  were  checked  against  the 
Trans-Optic®  sheets  and  the  referral  output 
data  to  ensure  that  the  data  file  values  were 
identical  to  the  original  data.  All  values 
from  these  random  samples  matched  hard¬ 


copy  values.  These  processes  are  consistent 
with  commonly  accepted  methods  of  data- 
handling  and  help  to  insure  the  integrity  of 
the  data  (Novick,  1985). 

Results 

Reliability 

Reliability  was  determined  by 
calculating  a  coefficient  alpha  (Cronbach, 
1951),  which  was  used  to  determine  inter¬ 
item  consistency  for  each  subscale.  For 
instance,  a  coefficient  of  .70  signifies  that 
70%  of  the  variance  in  scores  depends  on 
true  variance  in  the  trait  measured.  The 
preliminary  reliability  analysis  revealed  that 
22  of  the  25  subscales  demonstrated 
moderate  to  high  alpha  reliabilities  (alpha  of 
.5  or  above).  The  three  subscales  with 
inadequate  reliability  were  the  Self 
Criticism,  Problems  Submarine  School,  and 
Uncertain  About  Submarine  subscales.  This 
result  mirrored  Katz  and  Rexer's  earlier 
findings.  For  the  sake  of  brevity,  the 
preliminary  reliability  analysis  is  not 
presented  in  tabular  form.  After  an 
examination  of  the  item  content  and  the 
scoring  algorithm  was  conducted,  it  was 
determined  that  modifications  to  the  scoring 
procedures  were  needed  to  correct  for  the 
negative  item-to-scale  correlations  found  in 
these  subscales.  Additionally,  modifications 
were  also  needed  to  correct  for  negative 
item-to-scale  correlations  in  the  Social 
Isolate  and  Impulsive  subscales.  (These 
modifications  are  fully  described  within  the 
Problem  Items  section,  below.)  Following 
these  corrections  the  reliability  analysis  was 
performed  again.  Results  indicated 
moderate  to  high  reliability  for  all  25 
subscales,  with  alpha  values  ranging  from 
.65  to  .87  (see  Table  1). 
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Table  1.  Comparison  of  Means,  Standard  Deviations  and  Alpha  Reliabilities  by  Subscale. 


Katz  and 
Rexer 

Theriaque 

and 

Schlichtins 

SUBSCREEN  Subscale 

Mean 

Standard 

Deviation 

Alpha 

Reliability 

Mean 

Standard 

Deviation 

Alpha 

Reliability 

Distortion 

2.92 

0.38 

0.71 

2.96 

0.37 

0.73 

Good  Impression 

3.23 

0.31 

0.79 

3.24 

0.31 

0.81 

Mistake  Joining  Subs 

1.77 

0.44 

0.79 

1.76 

0.45 

0.80 

Uncertain  About  Subs 

2.37 

0.41 

0.54 

2.20 

0.51 

0.72 

Conditional  Submarine 

1.93 

0.49 

0.72 

1.89 

0.48 

0.74 

Unconditional  Submarine 

3.05 

0.46 

0.75 

3.09 

0,46 

0.77 

Problems  Submerging 

1.98 

0.45 

0.66 

1.93 

0.44 

0.68 

Problems  Sub  School 

2.40 

0.28 

0.03 

1.90 

0.42 

0.74 

*  Physical  Well-being 

1.68 

0.34 

0.73 

1.65 

0.34 

0.74 

*Low  Situational  Ctrl 

2.16 

0.38 

0.76 

2.11 

0.38 

0.77 

^Nervous  or  Worrying 

2.24 

0.37 

0.76 

2.19 

0.36 

0.77 

*Depressed  Mood 

1.81 

0.44 

0.86 

1.76 

0.44 

0.87 

*  Coercive  Attitudes 

2.05 

0.30 

0.71 

2.03 

0.30 

0.74 

*Aggressive/Destructive 

1.90 

0.36 

0.72 

1.87 

0.36 

0.73 

*Problems  Home/School 

1.90 

0.39 

0.76 

1.87 

0.39 

0.78 

♦Social  Isolate 

2.11 

0.32 

0.62 

2.07 

0.35 

0.70 

♦Impulsive 

2.31 

0.32 

0.67 

2.15 

0.34 

0.73 

*  Social  Support 

3.12 

0.37 

0.68 

3.20 

0.39 

0.70 

*  Unusual  Thoughts 

1.95 

0.38 

0.72 

1.90 

0.38 

0.75 

*  Unusual  Physical  Complaints 

1.62 

0.38 

0.77 

1.60 

0.38 

0.79 

*  Suicide  Subscale 

1.53 

0.42 

0.78 

1.51 

0.41 

0.79 

*  Claustrophobic  Feelings 

1.93 

0.40 

0.73 

1.89 

0.40 

0.76 

Problems  Nuclear 

1.71 

0.42 

0.63 

1.72 

0.42 

0.65 

Dependency 

2.34 

0.34 

0.68 

2.35 

0.33 

0.69 

Self  Criticism 

2.41 

0.26 

0.18 

2.19 

0.39 

0.73 

Note:  Bolding  indicates  improvement  in  reliability  of  subscales  where  items  were  reverse-scored  or  eliminated. 
*  subscales  used  by  the  scoring  algorithm  in  the  referral  process. 
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Problem  Items 

The  second  question  addressed  in  this 
study  related  to  the  nature  of  the  problem 
items  identified  in  Katz  and  Rexer's  1990 
study.  All  of  the  problem  items  noted  in  the 
earlier  study  were  re-identified  as  problem 
items  in  the  current  (initial)  reliability 
analysis.  As  noted  above,  this  confirmation 
led  the  researchers  to  further  examine  item 
content  and  scoring  methods  for  those  items 
that  correlated  negatively  with  other 
subscale  items.  Many  of  these  items  were 
found  to  be  correctable  by  reversing  the 
score  of  the  question.  The  rationale  for  this 
reverse-scoring  procedure  is  fully  explained 
in  the  Discussion  section  below. 

Specifically,  two  items  in  the  Problems 
Submarine  School  subscale,  three  items  in 
the  Self  Criticism  subscale,  and  one  item  in 
each  of  the  Social  Isolate,  Impulsive,  and 
Dependency  subscales  were  correctable  by 
adjusting  the  items’  scoring.  Adjustments 
were  made  by  recoding  subjects’  responses 
(A=D,  B=C,  C=B,  D=A)  for  the  items  in 
question.2  Of  the  five  subscales  listed  above, 
only  the  Social  Isolate  and  Impulsive 
subscales  are  used  as  referral  criteria  by  the 
scoring  algorithm. 

Problem  items  that  contained  an 
unusual  number  of  missing  or  neutral 
responses  were  determined  through  the  use 
of  Katz  and  Rexer's  criterion.  That  is,  any 


2  Note  that  subjects  initially  had  five  possible 
responses  ("A"  Strongly  Agree;  "B"  Agree;  "C" 
Neutral;  "D"  Disagree;  "E"  Strongly  Disagree), 
however,  during  the  scoring  process  all  items’  scores 
are  routinely  converted  to  a  four  point  scale  by 
removing  the  neutral  responses  from  the  scoring 
algorithm.  The  removal  of  these  neutral  responses 
took  place  before  the  recoding  of  the  subjects’ 
responses.  It  should  also  be  noted  that,  as  part  of  the 
typical  scoring  process,  alphabetic  labels  are 
converted  to  their  numerical  equivalents  (A=l,  B=2, 
C=3,  D=4)  to  allow  for  mathematical  computations. 


item  with  6.71  percent  (n=1310)  or  more 
missing  or  neutral  responses  was  considered 
a  problem  item.  Five  items,  contained  in 
three  subscales,  were  identified. 

Specifically,  two  items  in  the  Coercive 
Attitudes  subscale  (7.00%  and  8.53% 
missing/neutral),  two  items  in  the 
Unconditional  Submarine  subscale  (9.64% 
and  10.88%),  and  one  item  in  the  Uncertain 
About  Submarines  subscale  (8.2%)  were 
found  to  have  a  disproportionate  number  of 
missing  and/or  neutral  responses.  Of  these 
three  subscales  only  the  Coercive  Attitudes 
subscale  is  used  as  a  referral  criterion  by  the 
scoring  algorithm.  These  items  are  currently 
under  review  for  possible  re-wording  or 
elimination  from  the  inventory. 

Renorming 

The  third  issue  addressed  in  this  study 
involved  the  renorming  of  the 
SUBSCREEN.  As  is  standard  for  renorming 
procedures,  the  nearness  of  the  mean  of  the 
standardized  scores  to  zero  and  standard 
deviation  to  one  indicates  the  adequacy  of 
the  scoring  procedure  to  calculate 
standardized  values  (Katz  &  Rexer,  1990; 
SPSS,  1993).  Table  1  contains  the  new 
subscale  means  and  standard  deviations  (i.e. 
after  modifications  to  the  scoring  routine 
were  made)  as  well  as  Z-score  means  and 
standard  deviations  based  on  Katz  and 
Rexer's  values.  The  Z-score  means  and 
standard  deviations  for  all  25  subscales 
fluctuated  to  some  degree,  with  the  greatest 
deviations  found  in  the  Impulsive,  Uncertain 
About  Submarines,  Problems  Submarine 
School,  and  Self  Criticism  subscales  (see 
Table  2).  Of  these,  only  the  Impulsive 
subscale  is  used  as  a  basis  for  referral  by  the 
scoring  algorithm. 

While  no  formal  hypotheses  were 
considered  in  relation  to  the  renorming 
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procedure,  the  probability  of  observing  the 
calculated  standardized  means  was 
evaluated  through  the  use  of  a  normal 
distribution  table  (Howell,  1992). 
Probabilities  for  each  mean  were  determined 
by  examining  the  smaller  portion  of  the  area 
under  the  curve  for  the  mean  of  each 
subscale.  For  instance,  the  mean  of  the 
standardized  scores  for  the  Impulsive 
subscale  is  .98.  By  looking  up  this  value  in 
the  normal  distribution  table  we  find  that  the 
smaller  portion  of  the  area  beyond  the  Z 
value  is  .1635.  This  value  is  then  halved 
because  our  test  is  two-tailed  (.1635/2=.08). 
Thus,  we  can  say  that  if  we  take  a  random 
sample  of  standardized  scores  on  the 
Impulsive  subscale  from  an  infinite 
population  and  calculate  the  mean,  the 
probability  of  observing  a  mean  of  .98  is  .08. 

Discussion 

Cronbach's  alpha  was  used  as  the 
statistic  of  reliability.  Essentially  a  measure 
of  internal  consistency,  there  are  several 
interpretations  of  Cronbach's  alpha.  One 
involves  interpreting  the  alpha  value  as  "the 
correlation  between  a  test  or  subscale  and  all 
other  possible  tests  or  subscales  containing 
the  same  number  of  items,  which  could  be 
constructed  from  a  hypothetical  universe  of 
items  that  measure  the  characteristic  of 
interest"  (SPSS,  1993,  p.  147).  Cronbach's 

Table  2.  Z-Score  Means  and 


alpha  can  also  be  interpreted  as  the  squared 
correlation  of  the  observed  score  an 
individual  obtains  on  a  given  subscale  or  test 
and  the  subject’s  true  score  were  he 
questioned  on  all  possible  items  (SPSS, 
1993,  p.  147). 

Cronbach's  alpha  is  similar  to  a 
correlational  coefficient.  Its  values  typically 
range  from  0  to  1.  As  with  correlational 
coefficients,  the  nearer  the  absolute  value  is 
to  1,  the  stronger  the  association.  Typically, 
values  between  0.00  and  .39  are  considered 
"low,"  between  .40  and  .69  are  considered 
"moderate,"  and  between  .70  and  1.00  are 
considered  "high"  (Kerlinger,  1986; 
Henderson,  Morris,  &  Fitz-Gibbon,  1987). 

Based  on  the  determination  that  the 
scoring  of  several  SUBSCREEN  items  had 
not  been  properly  recoded,  a  brief 
explanation  of  the  rationale  for  recoding  is 
in  order.  Historically,  the  scoring  algorithm 
recodes  a  response  of  A  or  B  to  D  or  C, 
respectively,  if  an  answer  of  A  or  B  is 
considered  undesirable  or  problematic. 

Because  some  inventory  items  are 
worded  negatively  and  others  positively,  not 
all  responses  are  recoded.  A  simplistic 
example  will  help  to  clarify.  If  the 
SUBSCREEN  item  were  the  following:  “I 
feel  closed-in  when  I  am  indoors”  and  a 
student’s  response  was  A  or  B  (“Strongly 
Agree”  or  “Agree”)  recoding  of  the  response 

Standard  Deviations:  Problem  Subscales. 


SUBSCREEN  Subscale 

Mean 

Standard 

u 

Deviation 

*ImDulsive 

0.98 

-0.11 

0.080 

Uncertain  About  Submarines 

-0.43 

1.25 

0.170 

Problems  Submarine  School 

-1.83 

1.52 

0.015 

Self  Criticism 

-0.85 

1.50 

0.100 

Note  that  these  subscales  include  the  corrected  scoring  of  problem  items.  *  refers  to  subscale  used  by 
the  scoring  algorithm  in  referral  process. 
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would  be  in  order.  In  this  case,  agreement  to 
the  item  would  necessitate  a  higher  value  (4 
or  3,  respectively),  and  thereby  increase  the 
student’s  Claustrophobic  subscale  score.  On 
the  other  hand,  if  the  item  were:  “I  NEVER 
feel  closed-in  when  I  am  indoors,”  a 
student’s  response  of  A  or  B  would  not  be 
recoded  and  the  resulting  value  (1  or  2, 
respectively)  would  contribute  to  a  lower 
Claustrophobic  subscale  score. 

A  review  of  the  items  comprising  the 
three  subscales  noted  by  Katz  and  Rexer  as 
demonstrating  inadequate  reliability  re¬ 
vealed  that  the  scoring  for  several  items  had 
not  been  reversed  when,  in  fact,  it  should 
have  been.  Specifically,  two  items  in  the 
Problems  Subschool  subscale  and  three  in 
the  Self  Criticism  subscale  were  not  reverse- 
scored.  The  third  subscale  that  Katz  & 
Rexer  identified  as  having  inadequate 
reliability,  the  Uncertain  About  Submarines 
subscale,  suffered  because  of  the 
inappropriate  inclusion  of  a  single  item  that 
correlated  negatively  with  other  subscale 
items,  and  because  of  an  item  with  a  large 
number  of  missing  responses. 

As  presented  in  Table  1,  the  alpha 
reliability  of  these  three  subscales  fell  well 
within  acceptable  limits  once  the  scores  for 
the  problematic  items  were  reversed  and  a 
single  item  was  eliminated  from  the 
Uncertain  About  Submarines  subscale.3 

Several  other  subscales  needing  minor 
modification  were  noted  by  Katz  and  Rexer. 
The  difficulties  found  in  these  subscales  fell 
basically  into  two  categories  —  subscales 
with  items  that  correlated  negatively  with 
other  subscale  items,  and  subscales  with 
items  having  too  many  neutral  or  missing 


3  It  should  be  noted  that  the  removal  of  this 
item  from  its  subscale  occurred  after  Katz  and 
Rexer’s  analysis,  not  as  a  part  (or  consequence)  of  the 
current  investigation. 


responses.  Those  in  the  first  category 
included  items  found  in  the  Social  Isolate, 
Impulsive,  and  Dependency  subscales.  The 
scoring  for  one  item  in  each  of  these 
subscales  was  reversed  in  order  to  correct 
the  negative  correlation.  Those  in  the 
second  category  included  items  from  the 
Coercive  Attitudes,  Aggressive/Destructive, 
and  Unconditional  Submarine  subscales. 

The  items  from  the  subscales  in  this  second 
category  are  currently  under  review  and  may 
be  modified  or  excluded  in  the  future. 

Results  of  the  analysis  of  the  subscales 
that  suffered  due  to  missing  item  responses 
again  paralleled  Katz  and  Rexer's  findings. 
Two  items  in  the  Coercive  Attitudes 
subscale,  two  items  in  the  Unconditional 
Submarine  subscale,  and  one  item  in  the 
Uncertain  About  Submarines  subscale  were 
associated  with  this  problem.  While  an 
additional  13,000  individuals  have  been 
tested  since  Katz  and  Rexer's  1990  analysis, 
the  problem  items  previously  identified 
remain  a  source  of  difficulty.  However, 
neither  Katz  and  Rexer's  work  nor  the 
current  study  address  the  manner  in  which 
the  items'  content  is  questionable.  That  is,  it 
may  be  that  the  wording  of  some  of  these 
questions  is  outdated  or  obscure  (e.g.  "I 
believe  that  might  makes  right"),  or  simply 
that  some  of  these  questions  are  too  pointed 
(e.g.  "I  would  prefer  to  serve  under  a 
commanding  officer  who  is  easy"). 

The  adequacy  of  the  current  means  and 
standard  deviations  was  explored  by 
calculating  the  means  and  standard 
deviations  of  the  standardized  scores  and 
examining  the  probabilities  associated  with 
these  means.  Table  2  presents  those  data  for 
the  four  subscales  that  demonstrated  the 
greatest  deviation  from  1990  values:  the 
Impulsive,  Uncertain  About  Submarines, 
Problems  Submarine  School,  and  Self 
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Criticism  subscales.  The  probabilities 
associated  with  the  means  obtained  from 
these  subscales’  Z  scores  were  less  than  .2  (p 
<  .2).  This  finding  is  not  surprising  in  light 
of  the  fact  that  the  scoring  of  the  problem 
items  for  these  subscales  (the  items 
correlating  negatively  with  other  subscale 
items)  was  corrected  (reverse-scored)  before 
the  renorming  procedure  was  performed. 
This  alone  could  account  for  the  magnitude 
of  change  in  the  means  and  standard 
deviations  for  the  four  most  problematic 
subscales.  The  minor  fluctuations  found  in 
many  of  the  remaining  subscales  necessitate 
the  implementation  of  the  new  means  and 
standard  deviations  to  update  the  scoring 
process. 

The  use  of  the  new  means  and  standard 
deviations  is  not  only  supported  by  our 
empirical  findings,  but  also  by  the  literature 
as  well.  Good  test  construction  and 
maintenance  practices  dictate  that  the 
instrument  be  renormed  periodically  in  order 
to  ensure  continued  accuracy  in  the 
calculation  of  standardized  scores 
(Cronbach,  1990).  Further,  the  inclusion  of 
nearly  13,000  new  observations  to  the 
dataset  adds  meaningful  information 
towards  accurate  renorming.  As  Cronbach 
notes,  “When  the  sample  is  properly 
distributed,  a  larger  sample  gives  better 
norms”  (1990,  p.  128). 

Conclusion 

All  25  SUBSCREEN  subscales 
displayed  moderate  to  high  reliability  after 
corrections  were  made  to  the  scoring  of  a 
number  of  items  and  a  single  item  was 
omitted  from  one  subscale.  The  researchers 
found  that  all  of  the  problem  items  noted  by 
Katz  and  Rexer  fell  into  two  categories. 
Those  that  Katz  and  Rexer  identified  as 
being  negatively  correlated  with  other 


subscale  items  were,  in  fact,  simply  scored 
incorrectly.  The  C  code  scoring  routine  that 
is  utilized  on  a  weekly  basis  has  been 
updated  to  properly  score  all  240 
SUBSCREEN  items.  Documentation 
relating  to  the  changes  and  the  rationale  for 
them  has  been  incorporated  into  the  code 
and  each  response  sheet  will  be  annotated  to 
note  that  the  revised  scoring  method  was 
used. 

The  five  problem  items  that  were 
identified  in  1990  as  having  too  many 
missing  or  neutral  responses  were  confirmed 
in  the  current  study.  These  items  are  being 
reviewed.  They  may  need  to  be  re-worded 
or  perhaps  discarded  from  the  inventory. 

Beyond  the  fluctuations  found  in  the 
four  problem  subscales,  results  indicated 
that  small  variations  in  the  means  and 
standard  deviations  of  many  of  the 
remaining  subscales  have  occurred.  As  part 
of  the  continual  upkeep  of  SUBSCREEN, 
new  means  and  standard  deviations  for  all 
SUBSCREEN  subscales  have  been  included 
in  the  scoring  program.  Finally,  the  authors 
recommend  that  renorming  be  conducted 
biannually  to  ensure  continued  accuracy  in 
correctly  identifying  enlisted  men  at  risk. 
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